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1. ' Overview 

This project was a renewal of earlier work on Individualized Insrvuctior. 
for Data Access (IIDA) . Begun in July, 1976, with initial funding fc :. or.e 
year, the project was resumed in April, 1978, and is effectively corupIi..ted 
with this report and the final report which fllows. 

The project staff were divided into two :i-cups. The coniputer grcu: 
has been concerned with the design, implemeuL^iion and testing of the 
requisite computer progra^i?. These various programs have be:£n descri:, . 
in detail in an earlier report (1). Froi?. the user's point c view, th / 
are four major subsections of the total systein — three instru::tional 
exercises and the on-line search assistance mcde. 

By the end of February, 1979, the first and second exercises had under- 
gone system debugging. Preliminary formative evaluation was conducted in 
early March by various kinds of computer literate users actively looking 
for flaws in the system. With the second exercise providing the nucleus 
which was expanded into the assistance mode, both the third exercise and 
the assistance mode were ready for use in evaluation testing in the summer 
of 1979. 

The behavioral group of the project staff has been concerned with both 
formative and suramative evaluation of IIDA. In formative evaluation, the 
concern has been with monitoring systcpi development and providing feed- 
back and information for refinement and development of the system. In 
summative evaluation, the concern has been with the assessment of the 
impact and effectiveness of IIDA, and with the extent to which the objective 
of the project have been met. The preceding quarterly report (2) discussed 
the series of formative evaluations conducted with IIDA, and also indicated 
some of the kinds of changes made as a result of the evaluations. In 
addition, that report also dealt with one set of evaluations w^hich were 
both summative and formative in nature in that they were concerned both 
with the question of the impact of the system on the user and with modifi- 
cation of the system for future use. Both the results of these studies and 
some of the changes made in the sy :tem as a result of the studies are 
described in that report . 

The substance of this report deals with two studies conducted in the 
environment for which the system was intended. During the fall of L979, 
and the winter of 1979-1980, two field experiments were conducted with 



2 



res£!arcn ingin-, 
and Engl : rin .0: 
study — t per 
were cor. 2d 1: 
by the p ss 
the secor. ::\ - 
of IIDA t- ^ — 
who '^2Te "•:)' z:oi': 

IIDA as a ' ezi : 
was upon : ' II D. 
search tra -iig. 

Before i rinn: 
are scae ac. :w1cj. 
owed to P. Ded 
MacFarquhar. PI. 
Exxon Research a::, 
conducting the exp-: 
users of the r.xxon 
studies ver jond 
studies c, - not 



and T 
:pany 
-nce 



-aenists at tv - different r.xxon Researc: 



_a one of thes studies — ti.-:: ba-::line 
.:.z±ons of IIDA t_ ined and a. isi.d searchers 

s e r 3 v; ho had ha i searc.es :• : a r *: h em 
--'£.s at the Inf or:z:.t:icn C_nte: 0: le. Ir 
:..:ithod study — thr oerfor-anc^ a:_- reactic 
earchers were cor:p„red with ti: :■ s r = 

.r:'^ classroom t-aiariag and then .st. - 
- case the fccus as upon the 11 j } 
.4 information, and in the other c.^ :.>c 
r -i rials as a way or providing crL-i:r 



owed to Br " 
the studi' 
authors v; 
the IIDA 
puter tin. 
Research ai 
by DIALOG 
experiment:.:. 



.rian of the details af the two stud:-' ^ the e 
h should be made. First of all, th.iU . are 
^olbrook, M. K. Landsaerg, P. A. Loi^r.::. M. 
r.. Sheaton^ S. J. Swetaick, and K. R. alton jf 
' \ ii\ ig Cor}pany, who assisted in setting and 
: ■ i-air Thanks are also extended to the pa: icipant 
.nfrrr.c on retrieval services at the sites vnere the 
:ec . W. aout their cooperation and assistrace the 

lompleted. In addition, especial rhanks are 
-ace . rhe E:cxon Corporation. Without har initiative, 
. here ould never have been dene at all. ^inally^ the 
:o thank r^obert Rich of Princeton Universi:. ; who served 
a consultant on evaluation. The support : - the com- 
the studies described below was provided b* ihe Exxon 
igi jering Company, which paid the computer cosi: as well as 
.iOMIZNDEX, who allowed the use of their faciliti:, for the 



ara L. 
"•pc: ■ 

ret a: 



EKLC 



3 



2. Baieliae Stuc- 



The baseline stu^y vas condu 
Company facility at ^1. ^hara Park, 
study design cilled 
five searchers who 
f\ve comparable sea: 
on site- The nterr 



a profile of 
IIDA users, 
two groups of 
assistance pr 
modification. 
2.1 Pro 



Pla 

first IIDA c 
were made ir 
began the f 
December. 

2.1 

treatment ar 
assigned to 
group were l: 
on subjects 



r -rollect:. 

:rained 
::S were to 
^re was to . 
r: :. usage for 
::— icipated t ■ • 
•2 s would reve 
ich would t _i 
system. 

r the study at ; 

:r visited the 
:f 1979; the ac 

la October, ar 



i at the Exxon Res arch and Zngineeri.:. 
3w Jersey, in the fall of 1979. The 
ta on the sea^'^hos performed by zwe: 
-ssisted by II . In addition, tver' 
one by the ir. rmation retrieval st 
he intermedij. :ed searches to provi 
om "ison with z'.ie diagnostic usage c 
t aificant differences between th( 
ciences in the IIDA training and 
subsequently corrected by appropriate 



\am Park bG?.an Xvcll before the 
-lity. Prt^liminary arrangements 
on-site participant use of IIDA 
ricluded during the third week in 



1: i^n . The desigr 
:Gntrcl group experir. 
:h grcup. Those part 



EKLC 



- this study consisted of a simple 
• ral design with subjects being randomly 
.ipants who were assigned to the IIDA 
-ned :o search using :IDA programs and then did two searches 
their ovTi choosing. The other participants, in the inter- 
mediated group, submitted the next search topic arising from their regular 
work to the information center intermediaries. These searches were then 
conducted by the intermediaries through IIDA, but with all of the IIDA 
diagnostic messages being suppressed. Thus, records of the number of times 
the diagnostics were triggered were kept by IIDA, but the intermediaries 
did not get messages from IIDA. All of the searches were done in COMPENDEX. 

Selection of subjects . The staff of the library/information 
center at Florham Park sent out some 2,200 letters to the engineering, 
management, and professional level personnel on the site, briefly describing 
the opportunity to be involved in the IIDA bibliographic search training and 
inviting them to participate. Nearly 150 responses were received either by 
in-person, written, or telephone contacts. The names were randomly ordered 
so that date of response, level of position within the company, and 
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departnent and section would not be considerat io is for sel rtior.. From 
this randomized list, twenty-five participants were randoii: assigned to 
the IIDA group, and twenty-five to the intermediated group 

Those volunteers who were not selected to participate _:iiialiy were 
notified that they were being put on a list of alternates : :d vould be 
called if possible. Very early, it became clear that the - ^..:..ries of 
scheduling and syste:ii availability would make it difficult ; all of the 
participants to cor.pIc-ze the requisite portions of the stu.- 'ithin the 
time period allotte^ for the conduce uf the stu.iy; consequ two 
alternates were added to each group. By the end of the si: . , inter- 
mediated searches had been done for twenty-six participant , md twentv- 
four of those assigned to the IIDA training and assistance p-jup had 
completed the full course of training and searching. In a_l rases, the 
participants were unable to complete the necessary parts " ' he study as 
a result of scheduling problems or conflicts with travel o ither Ejvxon 
sites. 

2.1.3 ScheiuL'n^ the subjects . Considerable di :icu;Ly was 
encountered in scheduling the participants in the Florha" Par;: study 
resulting from the fact that the use of IIDA was limited Jurin?^ the fir-^t 
month of the study. This limitation was imposed by unexoectr-c hardv;are 
restrictions on the computer which housed the IIDA software, ar. a result 
of which, we were restricted to the hours between noon and 5:C. p.m. 
Since most of the participants left the site by 4:30, the hali-nour a^L-.e-- 
that was utilized by the intermediaries for completion of the searches 
submitted by the intermediated group. During the seoopd month of this 
study, the hardware problem was eased, and IIDA could be used for the 
entire day. Consequently, the restrictions on scheduling for the second 
month were those of staff and participant availability. 

During the first four to five weeks, during which the intermediated 
searches were done, the IIDA training group participants were the only 
ones scheduled for training. All participants were originally scheduled 
for two sessions. The first appointment was scheduled for one hour, during 
which it was expected that the participants would complete both exercises 
one and two. The i.econd appointment was scheduled for two hours, during 
which the subjects were expected to complete exercise three, and their two 
searches. This particular scheduling pattern was selected as a result of 
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expectations that tryir. do everything in one sitting was too much, and 

that the participants . find it difficult to fit more than two appoint- 

ments into their norm : schedules. Every attempt was made to schedule 

the first and second i - : inents within a period of no more than one week. 
In many cases, howev«^ :::ird appointment was necessary as a result of 
technical problems or _£ :_ result of either late arrival or early departure 
on the part of tne pa -:iiz_rants. 

2.1.4 Coadu,..! of the study . Each of the fifty participants was 
sent a pre-search que siionnaire before beginning the experiment, with 
identical questionnaires being used for each group of participants. The 
participants wore as'rv^i to return the completed questionnaires to the IIDA 
staff either by mail cr when they came in for their first appointment. No 
user was allowed to begin IIDA training without having returned a completed 
questionnaire. When ihe participant arrived. for the first appointment, the 
project staff took t;.- questionnaire and then filled out a card on the 
participant listing uaine, office phone and address, password, and the 
schedule of appointments which had been set up for the participant. For 
the intermediated participants, a card was completed at the time the inter- 
mediary completed the participant's search, which v/as done after the 
completion of the pre-search questionnaire. The card for these partici- 
pants listed the information mentioned above, as well as the password used 
in completing the intermediated search for the participant. The cards for 
both groups were color coded to indicate whether the participant was a 
member of the IIDA or the intermediated group, and were also used to record 
the portions of the training completed at each session, the amount of time 
it took to complete each exercise and each search, and to note any addi- 
tional appointments or any kind of help sought by the participant from the 
project staff rather than IIDA. Any peculiarities of the participant's 
IIDA experiences were noted on the card. 

The IIDA staff agreed in advance upon certain "rules" for monitoring the 
subjects' activities. Very little attention had been paid in the develop- 
ment of IIDA to creating a foolproof, user-oriented procedure for logging 
on to the system initially, or for recovering from DIALOG or telecommuni- 
cation crashes. Consequently, an IIDA staff member was physically present 
on-site for all IIDA training and searching sessions so as to deal with 
these problems. This solution to the problem of access to the system, 
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however, created other probleins in that with another person in the room, 
or nearby, some participants tended to want to use that person as a help 
library. The procedures established for dealing v;ith user questions 
directed to IIDA staff basically fit the follo^'ing hierarchy. If a 
participant asked for help from a staff Licnber, the staff r.eiuber made 
evc-iy effort to direct the participant to IIDA's help progra::^? to answer 
the question. If the participnat persisted and did not v;ant to use the 
help program, or did not know how, the staff nember would further encourage 
the use of help or briefly remind the user of how to get into the help 
prograni. \-Jhen all such efforts failed, the question w^as answered as briefly 
as possible and as directly as possible, and then a note was made of the 
incident on the participant's card. There were very few questions which the 
staff had to answer directly. Most of the questior^ directed initially to 
Che staff involved how to get started with the first practice search 
(exercise two), or fiow to decide w^hich optional material to take a look at 
in exercise three. These questions to the staff were easily deflected 
back to IIDA. The participants also asked frequc-ntly about the TYPE formats, 
but were easily led to the help program for a solution to the problem. 

The sequence of events for the IIDA trained participants was as follows: 
(a) complete the pre-search questionnaire; (b) complete exercises one, two^ 
and three, and the first search; (c) complete the iatcrmediate question- 
naire; (d) complete the second search; and. (e) complete the post-search 
questionnaire . The transcript of each participant ' s exercise one was 
given to him upon completion of the exercise. Transcripts of exercises 
two and three, and of the individual searches, were Xeroxed, and the 
originals returned to the participants. 

For the intermediated participants, the sequence was as follows: 
(a) complete the pre-search questionnaire; (b) iJubmit a search topic to the 
information center staff; and, (c) complete a "post-search** questionnaire 
about the results of the intermediated search. Transcripts of the inter- 
mediated searches were returned immediately to the participants along 
with the questionnaire about the search results. The post-search ques- 
tionnaire filled out by the intermediated participants consisted of the 
first page of questions from the intermediate questionnaire filled out by 
the IIDA trained participants. TTiese questions were duplicated in the IIDA 
group post-search questionnaire. Samples of the questionnaires used in this 
Q Study appear in Appendix A. 

ERIC 
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Since all of the participants were recruited with an offer of biblio- - 
graphic search training, those who were assigned to the intermediated group 
were given the opportunity for IIDA training after their intermediated 
searches, and the questionnaires about them, had been completed. Scheduling 
of training sessions for these users began early in the second month of the 
study. For the eighteen who actually completed the IIDA training, all 
records were kept, and they were also asked to complete the post-search 
questionnaire used with those participants who had been assigned to the 
IIDA training group initially. These data are not reported on herein. 

2.1.5 Data coding . Using a pre-established coding procedure, 
each questionnaire was coded, on the questionnaire, as it was completed by 
the participant. As each participant's folder of questionnaires and 
transcripts was completed, the folders were returned to Drexel, where the 
coding was checked by another staff member while the data was transferred 
to data sheets to be used for keypunching. Tnese sheets v/ere then checked 
by yet another staff member as the data cards were punched. Finally, the 
da-ta cards were checked against the original coding on the questionnaires. 
Diagnostic and search history data recorded by the IIDA record keeping 
programs were periodically listed out at MIT and sent to Drexel, where the 
errors, etc., diagnosed by the IIDA diagnostics were compared with the 
Xeroxed transcripts of the searches. This provided a check on the possi- 
bility of system or programming errors resulting in erroneous data 
recording. In the few cases where the data was lost from the machine 
records as a result of system failure, the data was taken directly from the 
search transcripts. U^ien all of this data had been transferred to coding 
sheets, the data cards were punched, proofed, and corrected if necessary. 
These cards were then checked against the original machine records and/or 
search transcripts, and corrected if necessary. 
2.2 Results 

Basically four kinds of data will be discussed below. These are: 

(a) demographic information collected on the pre-search questionnaires; 

(b) the frequencies with which the various categories of machine diagnostics 
were activated; (c) the various measures collected on the post--search 
questionnaire; and, (d) a description of the remarks made by the IIDA users 
on the post-search questionnaires. 

2*2.1 Demographic information . With the exception of two partici- 
^ pants, one in each group, all of those who took part in the Florham Park 
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study were engineers of one type or another. None of the participants had 
ever done on-line searching before, but alnost all had had on-line searching 
done for them at some tine prior to the beginning of the study, sonie as 
often as eight or nine tines. The overall average number of times the 
participants had had on-line searching done \:3s almost exactly three, and 
the two groups did not differ in this respect. AJ.l but five participants 
in each of the two groups knew at least one computer language. Tac two 
groups did no: lii'fer in the number of languages knou-^i, and the overall 
average was appr^^xiML^ily 1.5. In addition, the two groups did not differ 
in their ratings of i hoir level of computer skills, with the overall 
average rating being 2.8. The two groups also did not differ in their pro- 
search attitudes towards using the computer for work related projects. 
This was assessed using a series of six bi-polar adjective r^cales ^xored 
from one to seven, with one being least favorable. The average total score 
for each group of participants was approxmately thirty. 



providjn?, search assistance represent both ways of deciding when a user 



^' r^: \ics are a set of rules, clustered into categories, which are 

u.' y computer to decide which messages to send to the user, and 

when to i:-:] them. When a diagnostic routine detects a problem, IIDA 
intrudes on the user with a statement of what the problem appears to be, 
and with an indication of how to get advice or reference information. 
Thus the frequency with which a particular diagnostic is triggered repre- 
sents the number of times that a particular problem arose in the searching 
behav ioroftheuser. 

One type of diagnostic is concerned only with the validity of a 
DIALOG command, and is thus entirely context free. The guiding operational 
rule is that a command is valid if and only if it would be accepted by 
DIALOG. The second type of diagnostic— the local or individual command 
usage diagnostics — look at the most recent command in the context of the 
accumulated history of previous comraands. These rules deal with problems 
that range from fatal errors (such as the use of an undefined set number in 
a COMBINE command) to mere inelegance of usage (such as repeating a previous 
command). The specific sub-categories of this kind of diagnostic utiliz^ed 
by IIDA in this study include those that deal with: (a) repeated commands, 



2.2.2 Diag nostic s. The diagnostic procedures utilized by IIDA in 
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(b) creation of null sets, (c) unused sets, (d) uninf oraatlve print fortaats^ 
(e) excessive printing, and, (f) excessive time to enter a cocmand. 

The global, or ccmriand string diagnostics — the third major category — 
deal with a set of conamands as an entity. They are, in effect, concerned 
more with style than with mechanical exactitude. The first sub-category 
of this type of diagnostic consists of rules which deal with string and 
cycle length. Basically, a string of commands consists of one or more 
commands of the same type~e.g., SELECT, SELECT— and a cycle basically 
consists of seme number of strings of commands in which the user creates a 
set of references, takes a look at some of the members of that set, and 
then cycles back to refine that set. Generally, with novice users, an 
unusually long string of commands is an indication of a user problem. Two 
other ty-jes of diagnostics in this global category deal with either 
thrashing or dwelling. Thrashing involves changing the "direction" of the 
search rapidly or often, without any apparent progress toward a goal. 
Dwelling involves remaining with a search concept when it may well be time 
to give up and take another approach. The last sub-category of diagnostics 
in the global category deals with relevance. These diagnostics are used 
to direct the searcher's attention to the fact that a particular set seems 
not to be fruitful, or that a previously examined set yielded higher 
relevance scores. 

Before reporting the results of the comparisons betv/een the searches 
conducted by the intermediaries and those conducted by the IIDA trained 
searchers, there is an important point which must be discussed. As the 
result of an IIDA program system difficulty, some of the search records for 
the intermediated searches were not preserved by IIDA and printed out for 
later use. This problem was unrelated to the actual conduct of the searches 
by the intermediaries, but it was not discovered until many of the search 
transcripts had already been returned to the participants. Since several 
of the participants had already made use of, and disposed of, the search 
transcript, several sets of records were completely lost. Of the twenty- 
six search topics actually submitted to the intermediaries, IIDA diagnostic 
data were available on only sixteen of the searches. Since only sixteen 
sets of search records were available, the comparisons between the two 
groups on the diagnostic measures had to be conducted on fewer searches * 
than originally intended. In addition, the possibility exists that the 

ERIC 12 



10 



sixteen reraaining sets of diagnostic information were not representative 
of what the group of records would have looked like had the entire set been 
available. This raises a concern about whether or not it is reasonable to 
compare the diagnostic data of the intermediated and IIDA trained groups 
even though ten sets of diagnostic data are missing from the intermediated 
group. 

Ishile this question can not be answered with certainty, there are two 
factors which nake it plausible to assume that the pattern of results 
reported below does not differ appreciably from tha pattern that would 
have been found if the missing sets of data had been included. First, one 
of the participant identification numbers indicated the order in which the 
searches were done for the participants, and a Mann-\sTiitney U test was 
conducted comparing the tv;o groups of intermediated searchers, and treating 
the identification numbers as data. Second, the post-search questionnaire 
measures of those participants whose diagnostic data were missing were 
compared with the measures for those whose diagnostic data were not missing. 
For all but two of these latter comparisons, the Mann-Uliitney U test was 
used. For the other two — the two dichotonious variables — Fisher's exact 
test v/as used. Since it is not desirable in this case to have a difference 
between the two groups be undetected, the conservative alpha level of .25 
was adopted. None of the tests mentioned above indicated a significant 
difference between the two groups of intermediated participants. In fact, 
the smallest probability attained by any of the test results was greater 
than .32, so that it seems very unlikely that differences exist between 
the two groups of intermediated participants on the measures tested. 

The results mean two things. First, the problem which resulted in the 
loss of diagnostic data on some of the intermediated searches was unrelated 
to the order in which the searches were done. Second, the reactions of 
the participants to the search results produced by the intermediaries did 
not differ for the two groups who had intermediated searches done. Thus, 
the effect of the loss appears to be equivalent to the effect expected if 
the ten sets of diagnostic data had been dropped entirely at random. While 
this does not guarantee equivalence on the diagnostic data between the 
searches where the records are and are not available, it does imply that 
the comparisons reported on below, between the remaining intermediated 
searches and those done by the IIDA trained participants, probably would 
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have led to the same conclusions even if the diagnostic data for ten of the 
interaediated searches had not been lost as the result of a programming 



error. 



In looking at the comparisons between the searches done by the inter- 
mediaries and those done by the IIDA trained searchers, the pattern is one 
in which there appear to be no differences between the two sets of searches. 
The mean frequencies for each group of searches for each of the three major 
types of categories of diagnostics— syntax, individual command usage or 
"local" diagnostics, and command string usage or "global" diagnostics—are 
illustrated in Figure 1. For each of these classes of diagnostics, the 
differences between the two searches are not statistically significant 
(Mann-U'hitney U test, p > .05). This finding also holds true for each of 
the sub-categories of diagnostics. These data are summarized in Table 1 
for the intermediated searches, and in Table 2 for the searches done by the 
IIDA trained searchers. It should be noted that there is one sub-category 
of diagnostic information which is reported in Table 2 but is not included 
in the results illustrated in Figure 1, and which was not included in the 
comparisons conducted between the two groups. Since the intermediaries did 
their searches with IIDA suppressed, they vere not asked by the system to 
make relevance judgments on the references typed out. Consequently, this 
diagnostic category was not included in the data of the IIDA users when 
comparisons were conducted between the IIDA assisted and the intermediated 
searches. 

2-2.3 Post-search questionnaires . In the case of the inter- 
mediated group, the post-search data collected is based upon the first page 
of questions in the Intermediate Questionnaire in Appendix A. This one 
page was the questionnaire, along with a cover page, which these participants 
received when sent the results of the search which had been performed for 
them. The corresponding data for the IIDA trained group comer, from the 
identical questions which appeared in the Post-Search Questionnaire (see 
Appendix A), which was completed by these users after they had finished 
their second search. This set of questions includes five which ask the 
individual to rate his degree of satisfaction with various aspects of the 
search— amount of effort, amount of time, kind of assistance, items 
retrieved, and total search experience. Two other questions asked the 
participant to rate the usefulness of the results of the search, and the 
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e 1. Average frequencies for three major classes of 
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Dicignos tic 






ota. 






Category 


Total 


Avg. 


Dev. 


Median 


Range 


1. Syntax 


34 


2.12 




1 . JVJ 


u - o 


2. nonTTOnd 












Repetition 


1 


0.06 


0 25 




U — i 


j« uninj-Oiiiia Live 












Formats 


0 










4. Null Sets 












Created 


14 


0.88 


1.67 


0.17 


0-5 


5« Unused 












Sets 


3 


0.19 


0,54 


0.07 


0-2 


' 6. Time 


1 


0.06 


0.25 


0.03 


0-1 


7. Viewing Requests 












Excessive 


16 


1.00 


1.10 


0.75 


0-3 


8. String/Cycle 
Length 












9 


0.56 


0.89 


0.23 


0-2 


9. Thrashing 


0 


mm «• 








10. IX-jelling 


2 


0.12 


0.00 


0.03 


0-2 


11. Relevance 












12. Local (2 + 3 + 4 












+ 5 + 6 + 7) 


35 


2.19 


2.37 


1.83 


0-8 


13. Global (8+9 












+ 10 + 11) 


11 


0,69 


1.20 


0.23 


0-4 


Total (1 + 12 












+13) 


80 


5.00 


4.16 


5.50 


0-11 



'These diagnostics not operating for these searches. 

Table 1. Diagnostic data summary for Florham Park intermediated searches. 
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Diagnostic 
Category 


Total 


Avg, 


Std, 

Dev. 


Median 


Range 


1. 


Syntax 


35 


1.45 


1.38 


1.71 


0-5 


2. 


Cornnand 
Repetition 


1 


0.04 


0.20 


0.02 


0-1 


3. 


Uninf ormative 
Forraats 


2 


0.08 


0.28 


0.04 


0-1 


4. 


Null Sets 
Created 


37 


1.54 


2.02 


0.70 


0-7 


5. 


Unused 
Sets 


0 




M mm 






6. 


Time 


0 




mm mm 






7 


Excessive 


9 


0.38 


0.58 


0.25 


0-2 


8. 


String/Cycle 
Length 


20 


0.83 


1.68 


0.42 


0-8 


9. 


Thrashing 


0 




mm mm 






10. 


IX'7elling 


0 




mm mm 






11. 


Relevance 


43 


2.00 


2.28 


1.20 


0-9 


12. 


Local (2 + 3-1- 
4 + 5 + 6 + 7) 


49 


2.04 


2.10 


1.36 


0-7 


13. 


Global (8+9 
+ 10 + 11) 


68 


2.83 


3. 48 


1.50 


0 -14 




Total (1 + 12 
+ 13) 


152 


6.33 


4.67 


4.50 


0 - 18 



Table 2. Diagnostic data suminary for Florham Park IIDA trained 
searcher searches. 
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usefulness of tho sGarch as a v;holG. In addition, the participants were 
asked to decide v;hat percentase of the references retrieved v:ere: 
(a) Very Useful, (b) Useful, (c) Useless. For purposes of data analysis, 
these first tv;o categories were co::ibined into a single cat egory-~percontago 
useful. The renaining two questions asked the participant to indicate 
whcfther or not he had done any previous research on the search topic, and 
whether or not he would recoiuinend the system to friends. 

The data from tv;o of these questions—satisfaction with amount of effort 
and satisfaction with aniount of tiue involved— were not useable since 
several of the participants in the intern}edia t ed group chose not to ansv;er 
them. Comparisons were made, however, between the intermediated group and 
the IIDA trained group on the reinaining data. These comparisons were con- 
ducted on th. sun of the remaining three satisfaction rating scales, on the 
sum of the usefulness rating scales, on 'the percentage of useful references 
retrieved, on the number of users reporting previous research on the search 
topic, and on the nuniber of users who v7ould recoinrnend the search assistance 
systeia to friends. All coniparisons between the tv70 groups were conducted 
either with the Maun-Whitney U test. Fisher ^s e>:act test or the Chi square 
test . 

The two groups did not differ from each other in terras of their average 
total rated satisfaction and their rated usefulness of the search. Tho 
intennediated participants gave the searches done for them average satis- 
faction and usefulness ratings of 9.4 and 5.5 respectively, while the 
corresponding figures for the IIDA trained and assisted searciiers were 9.0 
and 5.4, respectively. In addition, the two groups did not differ 
significantly in terms of their judginents of the percentage of useful items 
retrieved. The interiaediated search participants indicated, on the average, 
that 49.3% of the references retrieved were useful, w^hile the IIDA searchers 
indicated that 52.5% of the references were useful. 

A total of 85% of the interniediated search participants were willing to 
reconrr.end that form of search assistance to their friends, while a total of 
88% of the IIDA trained and assisted searchers were willing to recommend 
IIDA to their friends. This difference is not statistically significant. 
In addition, with twelve of the intermediated and fifteen of the IIDA group 
having done sovue previous research on the topic of the search being rated, 
there was no difference between the two groups in terms of this measure of 
Q prior familiarity with the topic. 

IS 
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The ir.: rcorrelations of several measures from the post-search ques- 
tionnaire, diagnostic categories, and the pre-search questionnaire are 
reported i "able 3 for the interiDediated search group, and in ' e ' for 
the IIDA t: ..ne^- and assisted group, Not surprisingly, in botl: '.le 
individual ser's self-rating of computer skills is correlated \ 
year in which he received the Bachelor's degree. U^iat is not C: 
hov/cver, is why the IIDA trained searchers' pre-search attitude- d 
coraputer use in a job-related context should correlate signif icaaL l , ./ith 
rated coniputer skill and degree year, when no such correlations appear to 
exist in the interraediated group. However, even though the two correlations 
for the IIDA group are significant, they do not differ significantly from 
the correlations in the intermediated group. Thus, they may be a function 
of chance rather than of some systematic effect. 

The post-search questionnaire measures of satisfaction with the search - 
system, usefulness of the search, and willingness to recommend the system 
to friend^ all significantly intercorrelated for both groups. In 

addition, .2 correlations do not differ significantly from each other 
from one rrup to the other. However, the satisfaction and usefulness 
measures both correlate significantly with the judgment of the percentage 
of useful references retrieved in the IIDA group, but not in the inter- 
mediated group. Testing the differences between the correlations from one 
group to the other indicated a non-significant difference between the two 
groups for the correlation between usefulness and percentage useful, but a 
clearly significant difference between the two groups for the correlation 
between satisfaction and percentage useful (z = 2.89, p < .01). . This 
•suggests the possibility that the number of useful items retrieved is more 
important in influencing satisfaction with a search -.ssistance system when 
the user is actually spending seme time in doing the search. 

There are two other significant correlations in Tables 3 and A. For 
the intermediated searches done by the Exxon professionals, there is a 
correlation between local faults — individual command usage — and global 
faults — command string usage. For the IIDA group there is a correlation 
betw->^n ::he global faults and the individual's willingness to recommend 
the searih assistance system to others. However, neither of these corre- 
lations differs significantly from the corresponding correlation in the 
other table, and consequently they may well be a function of chance. 
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3 


A 


5 


6 


7 


3 


0 


10 


1. 


Computer Li;'.;c 


loOO 




















2. 


Coniputer Skill 


.20 


1.00 


















3. 


Degree Year 


.02 


.50V.-.V 


1,00 
















4. 


Satisfaction 


-.32 


.04 


.11 


1.00 














5. 


UscLiilne.ns 


-.23 


.06 


.07 


.60-v-;.- 


1.00 












6. 


Roconw.end^ 


-.32 


.03 


-.16 




.40^v 


1.00 










7. 


Glohal Faults^ 


,40 


-.03 


-.35. 


-.20 


-.27 


-.29 


1.00 








8. 


2 

Local Faults 


.03 




-.16 


-'.16 


-,17 


.02 


.49=v 


1.00 






9. 


2 

Syntax Errors 


.2G 


.41 


.17 


.19 


-.23 


.01 


.11 


.13 


1.00 




10. 


1 Useful Items 


.2G 


-.10 


.15 


-.09 


1 n 


.19 


.09 


.22 


-.26 


1.00 



*p < .05 
--■-^>p < .01 

■^Correlations involving this variable arc poirit-biserial correlation'^ 
1 . ' ' 

Correlations involvinc this variable arc based on an N of 16 rather than an N of 26. 

Table 3. Correlations for interRcdiated search group. 
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1. 


Computer Use 


1.00 


















2. 


Computer Skill 


.40* 


1.00 
















3. 


Degree Year 


AO* 




1.00 














4. 


Satisfaction 


-.12 


-.16 


-.18 


1.00 












5. 


Usefulness 


.15 


.13 


.14 


.72'Wt 


1.00 










6. 


Reconmiend^ 


.14 


.12 


.14 


.47* 


.39 


1.00 








7. 


Global Faults 


-.32 


-.16 


-.27 


-.05 


-.05 


.65'V-,v 


1.00 






8. 


Local Faults 


.29 


.22 


.26 


-.13 


.09 


.13 


-.07 


1.00 




9. 


Syntax Errors 


-.05 


-.07 


-.30 


.07 


.03 


.13 


.22 


.13 


1.00 


10. 


7. Useful Items 


-.31 


-.25 


-.14 




.51* 


.31 


.08 


-.24 


-.27 1.00 



^Vp < .05 
^v<p < .01 

^relations involving this variable are point-biserial correlations. 
Table 4, Correlations for IIDA trained search group. 
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2,2 J\ _Resp_ct!i scs of HP A use rs t o open- ended giicst ion s . On L h e 
post-search questionnaire filled out by the IIDA trained and assisted 
users, there v;ere questions which asked for written corraents about various 
aspecL-s of the search assistance syste^i. V/hile this information v/as not 
collected from the intermediated group, and consequently no direct com- 
parisons can be made, it is being presented here, nonetlieless , for its 
overfill value in interpreting the findings discussed above, 

a) Positive i mpress ions. Ifnen asked to give their positive 
impressions of IIDA, many users reported finding the overall system easy 
to use and to understand, noting in partici: t]\e fast response time. One 
user referred to lIDA's method of instruction as "simplified coaching," 
and another felt that the three initial exercises provided sufficient 
preparation to use the system satisfactorily. Some of the participants 
also appreciated the ability to select the degree of detail in the explana- 
tions in exercise three. The most frequent response was related to the 
availability of the help library, and IIDA's directions to specific areas 
of the help library follov/ing error messages. For example, one user 
reported that this feature saved him from making several additional errors. 
Also referring to the help features of IIDA, another user found the system 
to be, "...fairly forgiving, even mistakes are tolerated." Other positive 
impressions dealt with the co;rtmand summary at the end of exercise one, the • 
optional summary at the end of a search, and about IIDA*s tracking system 
for the relevance of the search. Ml but four of the IIDA users liad had a 
search done for them by the information center professional searchers at 
some time prior to the beginning of the study. Some of these users favored 
their experience with IIDA. As one user put it, "I can retrieve information 
quickly doing my own search because I know better what 1 am looking for.*' 
Others commented about the convenience of doing their ow^n searching, and 
about the elimination of the middleman." 

Several user comments w^ere directed tov;ard the positive aspects of on- 
line searching in general. Some users mentioned the simplicity of searching 
by creating sets in steps, which was helpful in narrowing one's search 
topic, and one user felt that EXPAND v;as especially helpful in this way. 
Another participant noted that the EXPAXD command w^as very useful in 
providing a source of ideas for alternative searches. Other features the 
users liked included the ability to retrieve descriptive abstracts and the 
Q ability to screen citations before asking for more details (various TYPE formats) 

ERIC 2'\ 
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b) Negatiive impressions . UTien asked to give their negative 
impressions of ^ IDA, several users reported having trouble remembering all 
of the instructions, and having difficulty in looking back over the printed 
exercises. Most of thea added they would have preferred a small "primer'' 
type manual with a brief summary of major steps, commands, and type formats. 
Some even felt this primer would have been just as good as IIDA on-line. 
Concerning IIDA's instructions, one user thought there should be more 
assistance ("coaching"), while another found IIDA "too verbose." 

A number of the responses dealt with the TYPE format and command. One 
user reported that TYPE format six gave too much information. Others sug- 
gested that there should be a means of terminating the printout on a 
citation if one finds the first part printed useless. A few users were 
dissatisfied with the limit on the number of citations which could be 
requested. T\';o of the participants expressed aggravation with the relevance 
ratings they were asked to make following each typed record. As one said, 
"It drives you crazy." 

Many of the comments dealt with various aspects of DIALOG or the data 
base. For example, several users appear to have had trouble selecting the 
correct descriptors in the data base to retrieve their specific requests. 
ITiey suggested that IIDA ought to provide an "automatic list of related 
terms," separate from the EXPAND command, as they appear in the file. Host 
of these users felt that additional assistance in this area would have 
helped them in narrowing down their searches. Other comments included: 
"Some articles didn't seem to fit the category for which I was searching," 
"Articles appeared which I thought I had already excluded from the final set. 
A few users complained that the combination of sets did not seem complete. 
Others disliked the importance placed on the order of the descriptors (e.g., 
"concrete blocks" retrieves different items than "blocks concrete), and the 
need to have separate sets for singular and plural forms of a descriptor. 
Only one user felt that the available data base was inadequate. 

Some of the participants were more concerned with the mechanics of the 
system, and found the print difficult to read, or the information printed 
too slowly for "first pass" searching. Several other negative comments 
concerned the mechanical or telecommunication problems which developed, such 
as, "...frequent inability to make connections, "...interruptions in program 
execution," and noise on-line causing the system not to accept all commands. 
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^) IlM.Aill]i^Jiron^g^jji_iiio. v.'hen asked to list situations 
in which IIDA did the wrong thinr., Lhe innjority of responses concerned tele- 
comnmnication problems such as: delays logging on, general noise on-line, 
disconnections froi. DIALOG in r.id-search , delays in receiving commands, 
and some corm.unicat i ons breakdowns resulting from improper responses, 'jn 
one case, IIDA was clearly at fault, in that the system ignored the user's 
request for a summary report at the end of the search: "It just logged off." 

d) Search_j^-.thout IIDA . If the participants felt thev had 
learned enough about searching that they could search on their own, they 
were asked to say when thoy first decided this. Of the sixteen who responded, 
three decided after er.ercise two; three reported deciding after exercise 
three; one, during the first search; five, after the first search; two. 
during the second search; and two, after the second search. 

Wien asked to explain their reasons for feeling they could search 
■ without IIDA assistance, most users felt that the only thing they would 
have difficulty with was the log on procedure, and getting hack on line in 
cases of telecommunication difficulties. Approximately one-third of these 
users found the search methods and application of commands easy to learn, 
and one user added that by the end of exercise three, one accumulates ' ' 
enough material and experience to search without assistance. A few parti- 
cipants reported that they could search without IIDA, but really were not 
familiar eaough with the vocabulary terms to do an efficient search. One 
user suggested that his search efficiency might improve with some exposure 
to the professional searchers. Although another user felt he could search 
v.'ithont assistance, he admitted that he would prefer a supplementary manual 
to, "reinforce what I learned from IIDA." Six of the users who responded 
either misunderstood the question (i.e., to search without assistance), or 
did not consider the assistance mode part of IIDA. They all reported they 
could search without assistance, but their explanations included relying 
on IIDA's help. Some of these responses included: "IIDA will help if 
needed," "...instructions are always available," "...if problems came up 
you could always ask for help," etc. 

Those users who did not feel that they could search without IIDA were 
quite uniform in their reasons. They explained that they needed more 
practice with commands and/or advanced techniques to be able to search 
with any efficiency. 
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2 . 3 Discussion 

In this study two raudonily assigat^d groups of users provided feed- 
back on the results of a bibliographic search. For one group of users, the 
search was done by a professional intemediary. The other group did their 
own searching with the assistance of the IIDA assistance programs, after 
having been trained in searching by the IIDA training exercises. Comparison 
of the two groups on several pre-search measures revealed no significant 
differences between the groups, indicating that the randomization had had 
the desired effect. The two groups also did not differ from each other on 
several post-search measures of their reactions to the searches. For 
e>:ample, in terms of judgments of the percentage of useful items retrieved 
in the searches, the difference was not statistically significar ITiis 
leads to the conclus ion that the IIDA trained and assisted users were able 
to do searches v/ith results equivalent to those produced by trained and 
experienced intermediaries. It was also found that there were no signi- 
ficant differences between the two sets of searches in terms of the number 
of IIDA detected errors or faults in the searching behavior of the pro- 
fessionals and in the searching behavior of the IIDA assisted users. 
Although comparable information was not available for both groups of users, 
the responses of the IIDA trained and assisted users to open-ended evalu- 
ation questions seemed to clearly indicate a positive response to the 
system by that user group. 

In light of the intent of this study, i.e., to provide a set of 
diagnostic benchmark criteria against which to assess the performance of 
the IIDA trained and assisted searchers, it was surprising to discover a 
lack of significant differences between the two groups of searches. IIDA 
was designed to produce acceptable results for its users, but it was never 
expected that the step-by-step performance would match that of professionals. 
A significant difference on one or more of the diagnostics would have 
pointed toward a deficiency of one kind or another in the IIDA training, or 
in the usefulness of the diagnostic messages during IIDA assisted searching. 
This would have led to revision of one or both aspects of the system. The 
finding of no significant differences also poses a potential problem in 
that this might suggest either that the IIDA training and diagnostics 
worked well, or that they are totally irrelevant. 

In looking at the lack of differences between the two sets of searches, 
there are some circumstances which suggest that the professionals may have 
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been somewhat handicapped relative to their habitual raode of searching. 
For example, the Exxon professional scarchervS are used to working with 1200 
baud terminals. With these terminals, more on-line searching is acceptable 
than with the 300 baud terminals used for IIDA searching. Hence, the pro- 
fessional staff v:as accustomed to doing a great deal of on-line printing, 
often exceeding the limit allowed for IIDA users working with the slower 
terminal. Mso, IIDA's design v;as fixed just before Lockheed announced 
the new SUPER SELECT command (3). Hence this command was not included in 
the IIDA training materials, and its syntax vzas not recognized by IIDA's 
parser. 

Although the intermediaries knew about these limitations, an inadvertent 
transferring of habits from their usual search context into the IIDA search 
context could have introduced some faults or errors which might not other- 
wise have occurred. For example, if a searcher used SUPER SELECT for set 
combination, then there V70uld be increased likelihood of the appearance of 
a largo number of SELECT commands, without any COMBINE command. This could 
then trigger an excessive string length diagnostic. Consequently, although 
it is not possible to assign numerical values, there is reason to believe 
that the error-fault rate of the professional searchers might have becin 
higher than normal. However, it is certainly the case that the IIDA 
searchers perform.ed respectably v;ell, and, in terras of the final outcome, 
the users who did their o\m searching using IIDA achieved results equivalent 
in utility to those v;ho worked through intermediaries. 

In fact, one reason for arguing that the training and diagnostic 
assistance routines worked as intended lies in the fact that the end user 
evaluation of the utility of the information retrieved in the searches did 
not differ significantly between the two groups. This means that a group 
of individuals v/ho had never before done on-line searching were able, with 
IIDA training and assistance, to do searches which produced results con- 
taining as much useful information as was contained in the results of 
searches done by professional searchers. They also did not differ signifi- 
cantly from those participants who had had intermediated searches done for 
thera in terms of either their degree of satisfaction with the search system 
used, or of their willingness to recommend the search system to their friends. 
Under these circumstances, the notion that the diagnostics are irrelevant as 
measures of searcher performance seems implausible, as does the notion that 
the training and assistance programs were ineffective. 
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3. Training Method Study 

The second major test of IIDA took place at the Exxon Research and 
Engineering Company facility in Linden, New Jersey, during the winter months 
of 1979-1980. This study involved a comparison of the searches done by IIDA 
trained and assisted searchers with those done by a group of searchers who 
were also assisted by IIDA, but who had been trained in a half-day class- 
room training session. The intent of this study was to test out the 
instructional procedures and materials involved in IIDA training against a 
more frequently used and more conventional training methods* Presumably a 
human instructor is more responsive to the problems which a student may 
encounter, and presumably the student has a greater range of questions 
which can be asked of a human instructor, especially since IIDA was not 
designed to be a question answering system. Consequently, it was antici- 
pated that significant differences between the two groups of searchers would 
reveal deficiencies in the IIDA training which would then be- subsequently 
corrected by appropriate modifications of the system. 

3. 1 Procedure 

Preliminary planning for this study was begun at the same time as 
the planning for the Florhara Park study. In fact, one of the contingency 
plans, which remained a plan only, was to use the Linden site as a back-up 
In the event of an unforeseen difficulty at the Florham Park site. This, 
however, was unnecessary, so during the third week of December and through 
the first week of February, users at the Linden site took part in an 
entirely different study th.in the one conducted at Florham Park. 

3.1.1 Design. The design of this study consisted of a simple 
two-trcatmant experiiaental design with subjects being assigned randomly to 
each group. Those participants; who wer^ assigned to the IIDA group were 
trained to search using the IIDA progracii?;. and then did two searches on 
subjects of their own choosing. The other subjects, in the conventional 
group, were trained to search in a classroom lecture-discussion format, with 
some time allowed for hands-on search experience, by one of the Linden 
information center professionals. They were then scheduled to do two searches 
of their own choosing through the IIDA assistance mode, with IIDA fully 
operational. All searches were done in COMPENDEX. 

3.1.2 Selection of subjects . The Exxon information center s taf f 
sent out a letter to the entire research staff at the Linden research 
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facilities, c:<plaining the opportunity to be involved in bibliographic 
search training, and inviting then to participate. The letter included a 
tear-off section for the volunteer to send back with name, section, office 
and phone number. It also listed the weeks during which the study was to 
run, one or more of v;hich the volunteer was to circle, indicating availa- 
bility. Nearly 150 responses were received. They were divided into groups 
by the inforraation center staff according to the weeks the volunteers were 
available for the study, and within these grouped according to the section 
in which the volunteer worked. From these groups, a stratified random 
saraple of fifty names was chosen, so as to obtain an even mix of partici- 
pants from different sections, and from those available during each of the 
six to eight wee'ris during which the study was expected to run. From this 
list of fifty, twenty-five subjects were randomly assigned to the IIDA 
group, and twenty-five to the conventional group. 

During the course of the study, it became necessary to make a few changes 
in these -,roups as a result of schedule conflicts, or of participants drop- 
ping out of the study. Tv;o of the participants, originally selected for 
the conventional group, were switched to the IIDA group because they were 
unable to attend the scheduled training session. In these cases, two 
participants, previously selected for the IIDA group, who were available 
for the conventional training session were switched to replace them. 
Throughout the course of the study, five participants (three IIDA and two 
conventional) dropped out as a result of lack of time or of travel require- 
ments. These participants were replaced from the remaining pool of volunteers 
by making a random selection from the slips for a particular week when a 
replacement was needed. Finally, using the same procedure as for replace- 
ments, one extra participant was added to each group about midway through 
the study simply to ensure that at least twenty-five participants in each 
group would finish in the time allotted for the study, mien the study was 
completed, there was, in fact, one participant in each group who had not 
finished the full course of training and searching. 

3.1.3 Scheduling of subjects . During the Linden study, the only 
restrictions on scheduling participants were the hours of staff and partici- 
pant availability. Generally, IIDA usage was almost continuous from 9:30 
a.m. to 4:30 p.m., Monday through Friday, from December 17, 1979 to 
February 6, 1980, with five days off over the weeks of the Christmas and 
New Year's holidays. 3(} 



All of the participants were scheduled for a total of two-and-one-half 
hours, including one-and-one-half hour of training and one hour of searching* 
However, vhile the IIDA group participants spent the full time on the IIDA 
system, the conventionally trained participants spent only the one hour of 
searching on the IIDA systeni. In general, the participants were scheduled 
for their various sessions a week or two in advance of actually beginning. 
All exercises and searches on IIDA were assumed to require approximately one- 
half hour each. The IIDA participants were all initially scheduled for 
three sessions, two for an hour each, and the third for a half-hour. The 
conventionally trained participants were scheduled for one training session 
lasting an hour-and-a-half , and then for two half-hour sessions on IIDA. 
The sessions were broken up this way so as to provide the participants some 
time between each to allov; them to assiniilate the information learned from 
each session. Each participant was originally scheduled so as to be able 
to complete the entire course of training and searching within three days 
to a week. However, some participants took longer than a week between 
starting and finishing. ITiese delays resulted from unanticipated schedule 
conflicts on the part of the participants, or from system problems — e.g., 
crashes, very long delays in getting logged on, etc, — which required 
rescheduling. 

3.1.4 Conduct of the stud y. Each participant filled out a pre- 
search questionnaire before beginning the experiment. These questionnaires 
were mailed out to all of the participants along with the letter informing 
them that they had been selected to participate in the study. The partici- 
pants either returned the questionnaires by mail, or brought them to their 
first session. None of the participants were allowed to begin the training 
without having completed a questionnaire. As with the Florham Park study, 
when a participant arrived for his first session with IIDA, a card was 
filled out on him with his name, phone and office number, IIDA password, 
and his appointment dates and times. The training session date for each of 
the conventionally trained participants was also noted on the card. 

The conventionally trained participants took part in one of four training 
sessions developed and conducted by the Exxon staff. These sessions were 
scheduled on the 17th of December, 1979, and on the 2nd, the 14th, and the 
21st of January, 1980. The classroom portion of each ran from 1:30 to 3:00 
in the afternoon, and the number of participants taking part in each session 
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ranged from five to eight. Although each session was conducted by a different 
professional, the same instructional materials and handouts were used in 
each. In addition, each session gave the participants some time on-line to 
practice the techniques they had learned. All of these participants were 
also given, either in the classroom session or in the brief introduction 
to IIDA on the system, a short explanation of the IIDA system, particularly 
the information on cancelling characters or lines with IIDA. The course 
handouts used in the conventional training sessions consisted of some forty 
•pages covering the various basics of on-line searching, including system 
commands, data base structure, and search strategy. Many of the pages 
presented examples of the effects of various commands, type formats, etc. 
In addition to this material, these users also had access, during training, 
to the DIALOG information on COMPENDEX, i.e., sample records, retrieval 
methods summary, and formats available summary. The conventional training 
sessions were structured to be roughly comparable in content and coverage 
to the IIDA training sessions (exercises one through three). A description 
of IIDA training materials can be found in an earlier report (4). 

The conventionally trained participants then performed two searches of 
their ovm choosing using IIDA. These searches were performed in the IIDA 
laboratory during two half-hour sessions. The amount of time actually 
spent by the participant for each search was recorded on his card, along 
with any comments about the system problems encountered, or questions which 
the participant may have asked of the IIBA staff member. After the first 
search, the participant filled out an intermediate questionnaire, and after 
the second search, a post-questionnaire. All of the questionnaires used 
in this study appear in Appendix A. 

The IIDA trained participants also came to three sessions, lasting a 
total of two-and-one-half hours, ideally, but their sessions were structured 
somewhat differently. In the first hour-long session, the participant com- 
pleted exercises one and two — the first tutorial exercise and a practice 
search. In the second session, also lasting an hour, the participant 
worked through exercise three — another tutorial exercise — and the first 
search of his oim choice. At the end of this session, the participant 
completed the intermediate questionnaire. For the last session, of half- 
hour length, the participr:nt again did a search of his own choice, and 
then filled out the post-search questionnaire. The time spent on each 
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exercise and search were recorded on the participant's card, along with any 
comments on system problems or questions asked by the participant. As in 
the Florham Park study described above, the IIDA staff made every effort 
to deflect user questions to IIDA rather than answer them directly. 

3.1.5 Data coding . The data coding and preparation procedures 
used in this study were essentially duplicates of those described above 
for the Plorham Park study. 
3.2 Results 

As with the previous study, basically four kinds of data will be 
discussed below. These are: (a) demographic information collected on the 
pre-search questionnaires; (b) the frequencies with which the various cate- 
gories of machine diagnostics were activated; (c) the various measures 
collected on the post-search questionnaire; and, (d) a description of the 
remarks made by the IIDA trained and by the conventionally trained users 
on the post-search questionnaire. 

3.2.1 Demographic inf orraat ion . A]. though one participant in the 
IIDA trained group did not have a B.A. degree, all of the other users had 
undergraduate degrees in fields with a scientific or technical orientation. 
Nineteen of the participants in each group had degrees in either Chemistry 
or Chemical Engineering. The remaining users had undergraduate majors in 
such fields as Physics, Biochemistry, and Physical Chemistry. Two parti- 
cipants in the IIDA group, and one in the conventionally trained group, had 
done some on-line searching for themselves before the beginning of the 
study. \Vhile five of the aIDA group, and six of the conventionally trained 
group, had never had on-lini^ searching done for them, most of the users had 
had searching done for them at some time before the beginning of the study. 
The IIDA group averaged 4.2 searches for those who had had searching done. 
The average for those in the conventionally trained group who had had 
searching done vzas 3.8 searches. The two groups did not differ on this or 
any of the measures discussed so far. They also did not differ in their 
pre-search attitudes toward using the computer for work-related projects, 
with each group having an average total score of approximately thirty. 

There was a difference between the two groups in terms of the number of 
computer languages knowm. The average for the IIDA group was 1.4, while 
the average for the conventionally trained group was .8 (Mann-Whitney U= 
208.5, p < .05). The difference, however, does not appear to be a partic- 
ularly strong one, since looking at only those in each group who knew one 
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or more languages indicates that these sub-groups did not differ, with 
an average of 1.9 for the IIDA group, and of 1.5 for the conventionally 
trained group. Further, while six of the twenty-five participants in the 
IIDA group, and twelve of the twenty-five in the conventionally trained 
group, knew no languages, the difference between the two groups is not 
significant. Turning to the individual's self-rating of computer skills, 
the two groups did not differ in their rating of their own computer skills, 
with the IIDA group having an average of 2.4, and the conventionally 
trained group, an average of 2.0. Consequently, there seems to be little 
reason to assume that the IIDA group had a clear advantage in computer 
skills, over the conventionally trained group. 

3-2.2 Diagnostics . In looking at the comparisons betv/een the 
searches done by the IIDA trained and by the conventionally trained users, 
again the pattern, as in the preceding study^ is one in which there appears 
to be no differences between the two sets of searches. The mean frequencies 
for each group of searches for each of the three major categories of diag- 
nostics are illustrated in Figure 2. It should be noted that the data in 
this study do include the sub-category of relevance diagnostics in the 
"global" diagnostic category, since both groups did their search with IIDA 
fully operational rather than suppressed, as was the case for the inter- 
mediated searches done in the study described earlier. For each of the 
three categories of diagnostics in Figure 2, the differences between the 
two sets of searches are not statistically significant (Mann-lNJhitney U test, 
p > .05). Furthermore, individual comparison of each of the sub-categories 
of diagnostics indicated no significant differences between the two groups. 
These data are summarized in Table 5 for the IIDA trained participants, and 
in Table 6 for the conventionally trained participants. 

3.2.3 Post-search questionnaire . For both groups of users, the 
post-search data collected is based primarily on the questions in the Post- 
Search Questionnaire in Appendix A. Participants in each group completed 
this questionnaire after having conducted their second of two self-selected 
searches. The questionnaire includes several sets of questions. One set 
of five asked the individual to rate his degree of satisfaction with various 
aspects of the search— amount of effort, amount of time, kind of assistance, 
items retrieved, and total search experience. Two other questions asked 
the participant to rate the usefulness of the results of the search, and 

ERiC 34 



3C 



X 

u 
en 
< 

CO 

or 

lU 
Q. 

CO 

i2 
< 

Ixi 



4.0 



3.5 



3.0 



2.5 



2.0 - 



1.5 



1.0 I 



0.5 



II DA 



□ 



CO.NJVENTIONAL 



SYNTAX LIMITED GLOBAL 
CONTEXT CONTEXT 



TYPES OF FAULTS 



Figure 2. 



Average frequencies for three major classes of faults 
at Linden. 
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Diagnostic 
o a L e g Ox i. e s 


Total 


Avg. 


Std. 
Dev. 


Median 


Range 


1. Syntax 


53 


2.12 


1.98 


1.43 


0-6 


2 , Cornmand 












Repetition 


4 


0.16 


0.37 


0.10 


0-1 


3 • Uninf onrta t ive 












Formats 


4 


0.16 


0.37 


0.10 


0-1 


4, Null Sets 












Created 


40 


1.60 


2.31 


0 39 


u — o 


5- Unuscid 












Sets 


1 


0.04 


0.20 


0.02 


n - 1 


6, Time 






1. 9 V 


0.04 


0-9 


7. Viewing Requests 












Excessive 


11 


0.44 


0.65 


0.28 


0-2 


8. String/Cycle 












Length 


14 


0.56 


1.29 


0.19 


0-6 


9. Thrashing 


0 










10. Dv/ellir.g 


0 










11. Relevance 


43 


1.72 


1.67 


1.57 




12. Local (2 + 3 + 












4+5 + 6 + 7) 


73 


2.92 


- 2.98 


1.44 


0-8 


13. Global (8+9 












+ 10 + 11) 


57 


2.28 


2.23 


1.86 


0-8 


Total (1 + 12 
+ 13) 


183 


7.32 


5.21 


7.00 


0-20 



Table 5. Diagnostic, data summary for Linden IIDA trained 
searcher searches. 
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Diagnostic 
Category 


1 OLclX 


Avg. 


Std. 
Dev. 


Median 


Range 


1. 


S>Titax 


68 


2.72 


2.03 


2.43 


0-7 


2. 


CoDrTiand 
Repetition 


3 


0.12 


0.33 


0.07 


0-1 


3. 


Unin forma t i ve 
Format 


I 


0.08 


0.28 


0.04 


0-1 




Null Sets 
Created 


61 


2.44 


3.44 


1.00 


0-14 


5. 


Unused 
Sets 


2 


0.08 


0.28 


0.04 


0-1 


6. 


Time 


6 


0.24 


1.20 


0.02 


0-6 


7 


Viev/ing Reques t s 
Excessive 


18 


0.72 


0.74 


0.65 


0-2 


8. 


String/Cycle 
Length 


32 


1.28 


1.93 


0.39 


0-6 


9. 


Thrashing 


0 










10. 


Dv/elling 


0 










11. 


Relevance 


37 


1.48 


1.61 


1.09 


0-7 


12. 


Local (2 + + 4 
+ 5 + 6 + 7) 


92 


3.68 


-3.79 


2.62 


0-15 


13. 


Global (8 + 9 + 
10 + 11) 


69 


2.76 


2.76 


1.67 


0-11 




Total (1 + 12 
+ 13) 


229 


9.16 


5.79 


9.12 


0 20 



Table 6. Diagnostic data sumnury for Linden conventionally trained 
searcher searches. 
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the usefulness of the search as a whole. In addition, each user was 
asked to decide what percentage of the references retrieved were: 
(a) Very Useful, (b) Useful, (c) Useless. Two other questions which 
also appeared on the first page of the questionnaire asked the user to 
indicate whether or not he had done any previous research on the search 
topic, and whether or not he would recommend the search system to friends. 

On the last page of the questionnaire appeared a set of questions 
which asked the participants to evaluate the search system by agreeing or 
disagreeing with a series of five statements. These items were scored on 
a scale from one to five, with five indicating a favorable evaluation of 
the systen on that item. Finally, the user was asked to rate the degree 
of understanding of the information retrieval system gained, and to 
indicate whether or not he felt that he could search on his own. 

Comparisons were made between the IIDA trained and the conventionally 
- trained group on these data. These comparisons were conducted on the sum 
of the satisfaction rating scales, the sum of the usefulness scales, the 
sum of the percentages of Very Useful and Useful items retrieved, the sum 
of the agree-disagree evaluation scales, the rating of degree of under- 
standing gained, the number of users who indicated they had done previous 
research, the number who indicated they would recommend the system to their 
friends, and the number who felt able to search on their own. In addition, 
the two groups of users were also compared in terms of the estimated time 
required to complete the second search. All comparisons between the two 
groups were conducted either with the Mann-Whitney U test. Fisher's exact 
test, or the Chi square test. 

The two groups did not differ from each other in terms of their average 
total rated satisfaction,- their rated usefulness of the search, and their 
gain in understanding of the system. The IIDA trained participants had 
averase satisfaction, usefulness, and understanding ratings of 14.3, 5.1, 
and 3.4, respectively, while the corresponding figures for the conventionally 
trained searchers were 15.0, 5.4, and 3.5, repectively. In addition, the 
two groups did not differ significantly in terms of their judgments of the 
percentage of useful items retrieved. The IIDA trained participants indi- 
cated, on the average, that 46.6% of the references retrieved were useful, 
while the conventionally trained participants indicated that 62.9% of the 
references were useful. 
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A total of 34% of the participants in each of the groups were of the 
opinion that they would be able to search on their own, and even more were 
willing to recommend the system to their friends, with 92% of the conven- 
tionally trained group, and SS% of the IIDA trained group, checking this 
question in the affirmative. Neither of these differences are statistically 
significant. In addition, with thirteen of the IIDA trained, and ten of 
the conventionally trained, having done some previous research on the topic 
of the search being rated, there was no difference between the two groups cn 
this measure of prior familiarity with the topic. A final measure on which 
there was no significant difference between the t-wo groups was the amount of 
time required to complete the search. At the time the searches were done, 
the IIDA assistant on site recorded an estimate of how long it had taken 
to do the search. The mean time to complete for the ilDA trained S^roup 
was 32.6 minutes, for the conventionally trained group, 30. 4 minutes. 

The two groups did differ, however, in their evaluation of the search 
system. The conventionally trained group was significantly more positive, 
with an average sum across the five scales of 19.6, than v.^as the IIDA 
trained, with an average sum of 17.8 on the five scales (Mann-V.liitney U = 
203.5, p < .05). Inspection of the scores on each of the five scales 
indicated that the conventionally trained group had given a numerically 
greater degree of agreement on each scale than had the IIDA trained group. 
The two groups were compared with each other on each item individually. 
The conventionally trained gave a significantly greater degree of agreement 
than the IIDA trained to the statement about the instructions being easy to 
follow (Mann-VJhitney U = 218, p < .05), and to the statement that the 
system was easier than expected to use (Mann-V.nitney U = 216.5, p < .05). 
Further, although the difference was not significant (Mann-V/hitney U = 
226, p < .07), the conventionally trained group appeared to give a higher 
degree of agreement with the statement that the search system was stimulating 
to use. 

The intercorrelations of several post-search measures, the diagnostic 
categories, and some of the measures from the pre-search questionnaire are 
reported in Table 7 for the IIDA trained searchers, and in Table 8 for 
the conventionally trained searchers. Some variables which might have been 
included in the Tables were omitted for various reasons. For example, the 
variable of previous research on the topic of the search was left out of the 
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1. 


Computer Use 


1.00 














2. 


Degree Year 


.17 


1.00 












3. 


Satisfaction 


.12 


.44* 


1.00 










4. 


Evaluation 


.27 


.47* 


.68** 


i.OO 








5. 


Understanding 


.43* 


.37 


.20 


.37 


1.00 






6. 


Time 


-.06 


-.18 


.21 


-.01 


.22 


l.CO 




7. 


Global Faults 


-.08 


-.32 


-.16 


-.07 


.18 


.41* 1.00 




8. 


Local Faults 


.10 


.06 


.05 


.12 


.51** 


.59'V* .30 


1.00 


9. 


Syntax Errors 


-.33 


-.28 


-.25 


-.30 


-.10 


.36 .36 


.18 


10. 


1, Useful Items 


.13 


.33 


.75*^v 


.52** 


.10 


.12 -.46* 


.06 



*p < .05 
**p < .01 



1.00 

-.13 1.00 



■11 



Table 7. Correlations for IIDA trained group. 
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2 


3 


4 


3 


6 


7 


8 


9 


10 


1. 


Computer Use 


1.00 




















2. 


Degree Year 


.03 


1.00 


















3. 


Satisfaction 


.49* 


-.04 


1.00 
















4. 


Evaluation 




-.15 


.60^bv 


1.00 














5. 


Understanding 


.37 


.12 


M-!< 


.35 


1.00 












6. 


Time 


.00 


.10 


.34 


.17 


.39 


1.00 










7. 


Global Faults 


.26 


.20 


.19 


.31 


.20 


.24 


1.00 








8. 


Local Faults 


.18 


.23 


.00 


-.19 


.08 


.06 


.09 


1.00 






9. 


Syntax Errors 


-.08 


.35 


-.21 


-.20 


.01 


.08 


.26 


.18 


1.00 




10. 


% Useful Items 


.12 


.13 


.43'V 


.33 


.29 


.26 


.12 


.04 


-.02 


1.00 



*p < .05 
< .01 

Table 8. Correlations for convGntlonally trained group. 
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Tables because of a lack of significant correlations with other variables. 
Similarly, the number of computer languages known, and the self-ratings of 
computer skill, which were highly correlated for both groups, were omitted 
because they shov/ed no significant correlations with any of the other 
variables investigated. Conversely, the judgments of usefulness were 
omitted because of their high correlations with, and only with, the 
satisfaction ratings score, and the other variables that satisfaction is 
correlated with in the Tables. Finally, willingness to recommend the 
system to others, and the user's belief that he could search on his ov/n 
were omitted because of the limitation on the value that the point-biserial 
correlation can assume when most of the responses fall into only one of the 
two categories on the dichotomous variable. 

While Tables 7 and 8 report several correlations that are significantly 
different from zero, those of major interest are the correlations where 
either the difference between the correlation in- one group and in the other 
is significant, or where each of the correlations is significantly greater 
than zero, and there is no significant difference between the two correla- 
tions. There are two pairs of correlations which fall into this latter 
category. In both groups of users, the satisfaction scores correlate 
significantly and positively with the evaluation ratings, and the judgments 
of the percentage of useful references retrieved. While the evaluation 
ratings correlate significantly with the percentage of useful items in the 
IIDA group, the correlation is not significant in the other group. In 
addition, the two correlations do not differ significantly from each other. 
If, however, the two correlations, not being different, are assumed to be 
two different estimates of the population correlation and are combined, the 
resulting estimate of the population correlation is significant (r = ,42, 
p < .05) • This pattern of correlations is compatible with some of those 
reported in the baseline study discussed earlier. There, the possibility 
was suggested that, for users actually doing their own searching, the degree 
to which the users are able to retrieve usable results influences their 
satisfaction with, and evaluation of, the search assistance system. * That 
suggestion would seem to find further support in the patterns of correlations 
described here, in that the same relationships seem to exist in the data 
from both studies. 
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In three of the cases reported in Tables 7 and 8, the correlation for 
one group differs significantly from that for the other group. The corre- 
lations between the year in which the bachelor's degree was earned and 
the evaluation of the search system differ significantly from one group 
to the other (z = 2.26, p < .05), as do the correlations between the time 
to complete the search and local faults (z = 2.13, p < .05), and between 
global faults and the percentage of useful items retrieved (2 = 2.10, p < 
.05). This latter correlation, which is significant for the IIDA group 
but not for the conventionally trained group, is probably the result of 
something specific to the sample of participants used in this study. 
These same two variables are not correlated significantly in the IIDA 
trained group described in the baseline study reported above. 

In the case of the first two correlations, where the IIDA group differs 
significantly from the conventionally trained group, both relationships 
seem to have no immediately obvious explanation, but may represent ideas 
worth following up in future research. For example, one speculation 
about the first correlation— between degree year and evaluation for the IIDA 
but not for the conventionally trained group — is that the IIDA group had 
considerably more exposure to the total set of IIDA programs, and the more 
recent graduates, having broader contact with and understanding of the 
strengths and limitations of computerized systems in general, were more 
tolerant or forgiving of the limitations of the system. 

3. 2. A Responses of IIDA trained users to open-ended ques tions. On 
the post-search questionnaire filled out by all participants, there were 
questions which asked for the user's written comments about various aspects 
of the search assistance system. This section summarizes, the responses of 
those users who were both trained and assisted by IIDA 

a) Positive impressions . When asked to give their positive 
impressions of IIDA, a large majority of the users directed their comments 
to various aspects of the assistance mode. Most of these users appreciated 
the help cues following the error messages. As one user explained it, "I 
had the feeling that if I goofed up the computer would 'know' how to get me 
out of it." Some of the other positive impressions about the assistance 
mode: the help mode allows one to operate more efficiently; help allows 
one to continue smoothly after making an error; the system did not "over- 
manage" the search; the error analysis was reasonable, and the "quick 
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information" notifies the user when he is on a dead-end route, or if the 
items requested are not in recent indexes. 

Several users were impressed with the rapid access to information, and 
IIDA's method of providing the user with "step wise acquisition of informa- 
tion." Others found IIDA easy to use, and found it easy to understand the 
command descriptions. A few users liked the way they were guided through 
the use of the commands, or the search summary option, or the relevance 
reports about the records retrieved. Overall, IIDA was described as doing 
"a good job of introducing techniques," with, "helpful instructions," and 
having assistance available. One user noted that with, "...little practice 
one could probably become quite proficient." UTiile another reported, "I 
would use this system often (if available)." 

Some participants' comments were directed toward DIALOG or on-line 
searching in general. Some of these favorable comments dealt with: 
working with unions and intersections of sets, the ability to change strategy 
in mid-search, the helpfulness of being able to expand tables, the desira- 
bility of being able to retrieve abstracts, and the immediate satisfaction 
from seeing the results on-line. Some of the mechanical difficulties 
which plagued the system showed up in the positive impressions in that 
one user noted that, "DIALOG was useful, when reached efficiently." 

t>) Negative impressions . When asked to give their nega- 
tive impressions of IIDA, the users' most f r equent r espons es concerned the 
limitations or irrelevance of the data base and the numerous telephone 
communication interruptions. Several users would have preferred a chemistry 
or physics data base. One user described the "service factor" as miserable 
because he returned six times for three appointments. Concerning IIDA's 
instructions, some of the participants found it difficult to assimilate 
all the information in the exercises, and would have preferred to have a 
supplementary primer for commands and short cuts. One suggested having 
shorter sessions, as he found it difficult to work on the system for long 
intervals. Other users would have liked more time on the system, speci- 
fically exercise three, to better understand its use for more efficiency. 
One user did not think that the system gave enough time to enter commands 
while the user was trying to remember them. 

Some users complained that IIDA's text was too conversational, had 
too many and too lengthy error messages, and felt that the system took too 
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long between responses-. According to some of the users, the relevance 
comments were a nuisance; they would have preferred not to give relevance 
ratings after every abstract. Questioning the validity of the relevance 
data, one user noted that one's opinion could change of a particular 
reference as one digs deeper into the subject. 

Several individual negative impressions were reported: confusion about 
which data base to use in exercise two, IIDA not being responsive to minor 
typing errors, difficulty in aborting a search if it is fruitless, and IIDA 
could have been more helpful with assistance on problems. 

Concerning DIALOG, some users found some items they retrieved were 
irrelevant to the selection. One user complained that the data base was 
poorly filed by author listings, while another found 'typos' in the file 
misleading. Confused by DIALOG'S filing system, one user misunderstood the 
difference between selected phrases and combined single terms. 

c) IIDA did the wronR thing . When asked to list the 
situations in which IIDA did the wrong thing, most comments referred to 
telecommunications problems: lost data from DIALOG connection requiring 
additional input, being logged off before it was requested, having the 
system print a statement five consecutive times. One user noted receiving 
no response from a command, then receiving the message, "DIALOG no longer 
responding to commands." 

The following are some individually reported situations in which IIDA 
did the wrong thing. One user began the search with /HELP. After the 
initial eight frames and instructions to choose a number of a DIALOG command, 
he entered /HL.52. This is not a DIALOG command, but IIDA returned the user 
to the main program just as if he had entered a DIALOG command. IIDA asked 
another user to type in a strange format number (ED/12126/90).. An error 
message appeared each time she tried to enter the number. Toward the end of 
a search, another user received the "illegal procedure condition" message 
for no apparent reason. One user mistakenly chose to search in file 201, but 
could not change files once she had discovered the problem. The system had 
trouble completing another searcher's references. He abandoned his efforts 
after several trials. Another user complained that IIDA was not "forgiving" 
of the spacing between a number and the command (e.g., BEGIN). Yet another 
user had trouble with the error messages. Three errors, with codes, appeared 
with instructions to type in a code number to get more information. None of 
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the coda numbers given the user exceeded 250. The user followed instructions, 
and received a message that acceptable codes included only those between 400 
and 500. These mostly represent software errors of omission, i.e., the 
system was not planned to respond in these ways, but obscure errors caused 
these peculiarities 

^) Search wit hout IIDA . The participants were also asked 
if they felt they could search on their own, and if so, when they had decided 
they knew enough to do so. Twenty of the users in this group responded 
with written comments. Two reported that they could not search without 
assistance. Four reported deciding they could search on their own after 
exercise two; one, after exercise three; and one. during the first search. 
The largest number of users, eight, reportedly decided after the first search 
was completed. One user decided during the second search; and two, after. 
Only one person seemed unsure, responding with a "maybe." 

When asked to explain their reasons for feeling that they could or 
could not search on their own, a few users felt they could as a result of 
gaining confidence from successfully completing a search in the assistance 
mode with limited assistance. Several of those „ho responded reported that, 
while they felt they could search without assistance, they needed a better 
summary of commands, or that they could search without assistance only on 
familiar topics. Those who felt they could not search without assistance 
listed their reasons as not feeling comfortable enough with advanced com- 
mands, or needing more details on search strategy, or simply needing more 
experience. One user did not think of IIDA as being more than the instruc- 
tional exercises, apparently, since he answered that he could search on his 
own. "...with help aids." 

3-2.5 Responses of conventionally trained users to open-ended 
questions . This section of the report summarizes the responses of the con- 
ventionally trained, but IIDA assisted, users to the questions on the 
post-search questionnaire calling for written responses. 

Positive impressions . When asked to give their positive 
Impressions of IIDA, most of the users in the conventionally trained group 
actually responded to characteristics of DIALOG rather than of IIDA. The 
largest number of responses about IIDA concerned the help library and the 
directions given by IIDA following syntax errors. Other responses referred 
to IIDA's instructions. The comments included: "self-explanatory system," 
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"the language is clear," "the system guides the user," "commands were 
simple," and, "(the system) was enjoyable to interact with." 

Overall, the majority of the responses were positive impressions about 
the time-saving aspect of searching on-line without an intermediary, and 
the quick responsiveness of the system. Several individual impressions 
about DIALOG referred to such things as: keeping track of sets accumulated, 
expand tables offering options to create new sets, having the ability to 
scan titles quickly, selecting with infixes, combining and sorting sets, 
and changing commands easily. One user found the system very easy to use, 
straightf ora^ard, and reported that, "It did nothing unexpected." 

b) Negative impressions . asked to give their negative 

impressions of IIDA, many of the users again responded to characteristics of 
searching in general rather than to IIDA. Some of the IIDA-specif ic comments 
included one user who was dissatisfied with the time limit to either enter 
a command or be logged off. Another thought the comments made by IIDA 
were too lengthy and complicated, and a second reported having trouble 
interpreting them. Three of the users were dissatisfied with IIDA's 
limitation on the number of records which one is permitted to view with 
each use of the type command. An additional complaint was made by one user 
who felt that there was not, and should be, an alert for such simple 
problems as improper use of the carriage return. 

Overall, one user thought the system too complicated, and some of the 
others thought the whole process of searching took too long, or at least 
longer than expected. Others referred to the delays and problems with 
getting on the system, and the pauses that occasionally occurred throughout 
the search, as a waste of time» Also related to the use of time, one user 
felt that his time could be used more effectively by doing a preliminary 
literature search followed by having an exhaustive search performed by a 
professional searcher. One user would have liked to have had the option of 
breaking from a print-out of the abstracts after having looked at the first . 
few lines. 

In doing the search, some users had trouble selecting key words, since 
there seemed to be a mismatch between the user's descriptors and those in 
the file. This problem caused some searchers difficulty in narrowing down 
the search. Several users were dissatisfied with the selection of available 
data bases, and a few were unhappy with the data base used. For example. 
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one reported misspellings filed in the data base, and another was dissatis- 
fied to find Exxon production research not included in the corporate sources. 

1 1 DA did the wrong thin?. , l-^en asked to list the 
situations in which IIDA did the wrong thing, users reported a wide variety 
of problems. Some of these problems seem to be related to IIDA specifically, 
but many of them appear to be based upon telecommunications difficulties 
and related problems. The kinds of problems users noted included: delays 
in logging on, being logged off too soon, having to return for numerous 
sessions because of system communication breakdowns, the system not accepting 
a command, receiving "erroneous results which neither I nor the IIDA assistant 
understood," losing the response to a SELECT, and being rejected when trying 
to re-enter the command, and useless repetitions as the result of "noise." 
One example of what may have been an on-line noise problem involved selecting 
a particular descriptor and receiving a null set. When the same descriptor 
•was selected again, it retrieved several items. 

d) S earch without IIDA . The participants were also asked 
if they felt they could search on their own, and if so, when they had 
decided they knew enough to do so. Twenty-one of the conventionally trained 
users felt they could search without assistance. Three reported first 
deciding this after the conventional training course, and one had already 
decided before the training began. Two of the users decided during their 
first assisted search, and seven, after. Another six decided during the 
second search, and two reported deciding after the second search. The major 
reason for feeling they could search without assistance was the simplicity 
of the commands. Some thought they would still use_ the course training 
manual. One thought he could searca without any further assistance or 
reference, but not in the engineering file or in fields outside of his 
interests. 

The four users who felt that they would be unable to search without 
assistance reported that they needed help defining the search terms, or in 
finding appropriate terms. One user thought that he simply needed more 
experience. 

3.3 Discuss ion 

In this study, two randomly assigned groups of users provided 
feedback on the results of an on-line bibliographic search. Both groups of 
users did the search themselves with the assistance of the IIDA on-line 
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assistance programs. One group of users had been trained in searching by 
the IIDA instructional exercises, while the oth^ir group of users received 
their training by a more conventional training method. Comparison of the 
two Kii}U|i:i uti ML'Vcrnl pre-search measures revealed only one significant 
difference between the two groups. This difference, in the average number 
of computer languages known by the users in the the two groups, did not 
seem to be a very strong one, however, in that further investigation 
indicated that the nunber of users who knew no languages at all was not 
significantly different between the two groups, and that those users who 
knew one or more computer languages did not differ between the two groups 
in terms of the mean number of languages known. Further, the number of 
languages knovm turned out not to be correlated with any other variable 
investigated in the study except the individual's self-rating of computer 



The two groups also did not differ from each other on several post- 
search measures of their reactions to the searches and the search 



of useful items retrieved in the searches, the difference was not statis- 
tically significant. Isfhere differences did exist, they were related to 
characteristics of the search assistance system rather than the search 
results. Those users who were conventionally trained seemed to be more 
likely than the IIDA trained searchers to agree that the instructions in 
the search system were easy to follow, and that the system was easier to 
use than had been expected. Further, the conventionally trained group 
expected to encounter more difficulty with the system than was in fact the 
case. These differences reflect favorably on the IIDA assistance programs 
in that experience with the IIDA training exercises is clearly not neces- 
sary for the individual to be able to make use of the assistance mode. 

It was also found that there were no significant differences between 
the tv/o sets of searches in terms of the number of IIDA detected errors or 
faults in the searching behavior of the two groups. Finally, although 
statistical analyses were not conducted, the responses of the two groups of 
searchers to open-ended evaluation questions seemed to clearly Indicate 
two things. The first is that IIDA is not a finished product without fault. 
The questions designed to elicit negative comments from the users were 
clearly able to do so. The second is that, overall, most of the users 



skills . 



assistance system. 



For example, in terms of judgments of the percentage 
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seemed to like learning to search, either with or without IIDA, and they 
seemed quite positive about the IIDA assistance they received. It is also 
interesting to observe that the users did not draw a clear distinction 
between the characteristics of IIDA and those of DIALOG or the telecommun- 
ications. This is not surprising, since IIDA was designed to be as 
unobtrusive as possible during searching. However, system designers 
should be aware of the fact that users generally tend to evaluate the system 
as a whole, and that the newly designed component, such as IIDA, may get 
either the credit or the blame for characteristics of the other parts of 
the system. 

In light of the intent of this study, i.e., to compare IIDA training 
with more conventional training, with a view to refinement of the IIDA 
programs, it was surprising to discover so few significant differences 
between the two groups of searchers. Further, the two groups, trained by 
- different methods, did not differ appreciably either on the diagnostic 
measures or in the percentage of useful references retrieved. A signif i- • 
cant difference on one or more of the diagnostics would have pointed toward 
either a deficiency in the IIDA training materials, or in the usefulness of 
the diagnostic messages during assisted searching. This would have led to 
a revision of one or both aspects of the system. The original design 
specification for IIDA did not envision a system to be competitive with 
more conventional training approaches . Rather, the intent was to provide 
an avenue of access to on-line searching for individuals who could not or 
would not do a conventional training course, and who still wanted personal 
rather than intermediated access to a data base. 

The finding of no significant differences on the diagnostic measures 
could pose a problem for interpretation of the results were it not for the 
fact that both groups were able to retriev-:^ a significant percentage of 
useful references during their searches. Presumably this represents some- 
thing which neither group would have been able to accomplish if simply 
turned loose with a terminal, but without having any training or assistance 
of any kind. Further, the two groups did not differ in their estimates of 
the percentage of useful references retrieved. This pattern of results 
argues strongly for the idea that IIDA training , as it is presently 
structured, represents a viable alternative to the type of conventional 
training with which it was compared. One reasonable guess as to the reason 
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for the pattern of results reported here is that, while the human instructor 
may well in facu have been more flexible and responsive in assisting the 
student and in answering questions during training, the design of ITDA does, 
as intended, enable the user to discover where and how to go get the 
information needed to answer questions for himself. 
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4. General Discussion 

A major goal of this project has been to provide a method for allowing 
direct user access to bibliographic searching. Thus, the atteirpt has been 
to develop a set of computer software packages which can provide on-line 
assistance to occasional users of retrieval systems. This collection of 
programs is also intended to be able to provide instruction, if needed, in 
the commands used in searching and in search strategy. When originally 
conceived, the expected utility of IIDA lay in the area of what might be 
referred to as "problem solving searches," These are searches where the 
end user of the information does not know exactly what the characteristics 
are of the desired set of references until they have actually been found- 
Consequently, it is very difficult for the end user to describe the prob- 
lem to an intermediary. There is no reason, however, why the IIDA user 
could not and should not make use of the system for all kinds of searches, 
if desired. 

The primary intended IIDA user has been the working scientist or 
engineer who may need access to the data base only a few times a year, 
and consequently is not interested in training oriented toward those who 
become professional iatermediaries. This person is assumed to be com- 
fortable using computers, but not necessarily trained in their use. In 
addition, it is assumed that this user is a serious, well-intentioned 
searcher who is trying to use the system to solve a problem- The IIDA 
software and diagnostic procedures were created to help remove the barriers 
to access for these users. Minor modifications of the system, however, 
could make it available to a much wider audience. 

Given the nature of IIDA, it is not possible to completely separate the 
performance of IIDA from the performance of the IIDA users. Looking first 
at the baseline study, the data suggest that IIDA users did searches which 
were, on two kinds of measures, done as well as those done by the more 
conventional approach of having the searches done for them by a professional 
searcher. First, in the judgments of the percentage of useful references 
retrieved in the searches, there was no significant difference between the 
results obtained by the professional searcher and those obtained by the 
user performing his own search. Second, it was found that there were no 
significant differences in the number of IIDA detected errors or faults 
between the newly trained, IIDA assisted searchers and the professional 



A8 

searchers. Together, these findings imply that the results of the searching 
were about equal, and that the performance of the searchers was about equal. 
In addition, the average rated satisfaction of the IIDA users and the parti- 
cipants who had intermediated searching done did not differ, and the majority 
of the users in both groups were pleased enough with their results and 
experience that they were willing to recomnend the system to their friends. 

The two different user groups which differed in the method of training 
prior to doing IIDA assisted searching also did not differ appreciably 
either in the diagnostic measures or in the percentage of useful references 
retrieved. In addition, the members of both of these groups did not differ 
significantly trora each other on several measures of their reactions to the 
search system, with, for example, most of the participants feeling they 
would recomraend the system to their friends. UTiere the two groups did 
differ, however, was in the evaluation of the system instructions and the 
ease of use of the system. In both cases, the conventionally trained users 
indicated a higher degree of agreement than did the IIDA trained users. 
That is, they were more likely to feel that the instructions were easy to 
follow, and more likely to indicate that the system was easier to use than 
they had expected. 

The finding of so few significant differences, particularly on the 
diagnostic measures, came as somewhat of a surprise, in that IIDA was 
designed to produce acceptable results for its users, but it was never 
expected that the step-by-step performance of the users would match that 
of professionals. Also, the original design specifications of IIDA did 
not envision a system which would be competitive with more conventional 
training approaches. Rather, the intent was to provide an avenue of access 
for individuals who could not or would not do a conventional training course^ 
and who still wanted personal rather than intermediated access to a data 
base. 

In a case where no significant differences exist, the situation is 
always troublesome in that such a finding could occur because the two groups 
did not differ, or because the measures chosen were irrelevant. This latter 
explanation, however, seems to be less reasonable than the idea that the 
system is working as it was designed to work. One reason for arguing that 
the training and diagnostic assistance routines worked as intended lies in 
the fact that in the baseline study the end user evaluation of the utility 
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of the information retrieved in the searches did not differ significantly 
between the two groups. This means that a group of individuals who had 
never before done on-line searching were able, with IIDA training and 
assistance, Co do searches which produced results containing as much useful 
information as was contained in the results of searches done by professional 
searchers. A second reason for not assuming that the diagnostics are 
irrelevant as indices of the searching behavior of the various user groups 
lies in the fact that the diagnostics were all empirically developed. 
That is, they were all designed to index and deal with problems encountered 
by searchers which have been either observed by the designers, reported in 
the literature by others, or both. 

However, recognizing that the conclusions drawn about the effectiveness 
of the diagnostics are not as clear and unambiguous as might be desirable, 
there are some additional directions for future research which ought to be 
• followed. The first of these has to do with the evaluation of the diag- 
nostics. The overall requirements for evaluation of the project as a 
whole, as well as the available resources, dictated that some studies have 
a higher priority than others. This led to a concentration on the 
evaluation of the IIDA system as a system rather than a more direct focus 
on the diagnostic component of the system. A future evaluation devoted 
solely to the question of the performance of the diagnostic system, with 
and without various kinds of training in advance, would be both useful 
and desirable. 

Another line of investigation dealing with the diagnostics which could 
be followed has to do with the possibility of adaptation to user skill 
level. The diagnostics were designed for use in a limited context: the 
training and assistance of the kind of novice user described earlier and 
in the user studies reported here. There :jas some indication in the 
experiences of the professional searchers that, when the diagnostics are 
employed with highly experienced searchers, they may not be the most 
appropriate procedures. There may have been times when the diagnostics 
were triggered by relatively sophisticated searcher behavior which is 
appropriate in one context, but not in situations with which IIDA was 
designed to cope. Thus, there seems to be no reason to assume that the 
same diagnostics will be equally useful or desirable for all user skill 
levels. Consequently, the diagnostics should be tested out more thoroughly 
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with persons of differing skill levels. In addition, it would also be 
desirable to test different variations of the diagnostics. In particular, 
it would be of interest to deternine whether or not it is possible, by 
varying the thresholds in the existing set of diagnostics as a function of 
the prior performance of the user, to get the diagnostics to perform 
adequately with individuals of different skill levels. That is, can they 
be made adaptive? 

A third direction for future research has to do with the type of IIDA 
user. Although the original target user group for IIDA consisted of 
technically trained individuals interested in a particular class of search 
problems, there now seei:ns to be no reason not to attempt to extend IIDA. 
The technically or scientifically trained user may be only one of several 
kinds of users who would find IIDA attractive and useful. In particular, • 
it seems desirable to determine whether or not a system such as IIDA can be 
used to provide direct data base access for a w-ide variety of possible end 
users interested in a wide variety of search problems. In fact, it may 
well be that IIDA, or an IIDA-like system, is a viable method of intro- 
ducing the computer and on-line searching to users with little or no 
background with computers. 

The fourth recommendation for future research follows from one of the 
limitations of the studies described above. These studies, and many other 
studies on searcher behavior, tend to be flawed by the fact that there is 
a relatively short amount of time between when the participants in the 
testing are first exposed to the system, and the evaluation of their 
performance, or the syctem performance. In addition, there seems to have 
been little or no testing of information system users at several points 
over a long period of time in the investigation of development or change 
in behavior over time. In the studies described above, the volunteer 
subjects generally tended to want to do the whole set of exercises, and 
then do their searching in a relatively short time . It seems particularly 
desirable that a long-range study be done, over a period of a year or so, 
in which the focus is upon how different people adapt to a new system, how 
quickly they adapt, how their behavior changes over time, and how it changes 
as the result of multiple search experiences. 

In conclusion, it seems that a new idea has been fairly tested in the 
very environment for which the concept was intended, and has come off quite 
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well. Indeed, one of the important characteristics of the two studies 
reported here is that there appeared to be no differences among the results 
produced by the various user groups when there was every reason to expect 
in advance that there would be a number of differences- While some aspects 
of these results are not entirely conclusive, they are supportive of the 
idea that the IIDA diagnostic procedures did indeed adequately measure 
important aspects of user performance- What is more certain, however, is 
that the IIDA system represents a way of training and assisting novice 
users in doing their own data base searching on problems of their own 
choice, with a level of performance that matches that of more experienced 
searchers- Furthermore, IIDA clearly represents a viable alternative route 
to gaining direct data base access for those end users who can not or will 
not do more conventional forms of search training- 




52 



5, References 

1 . Individualized Ins tiruction for Da ta Access , Quarterly Report: No . 6» 
NSF Grant No. DSI 77-2652A, School of Libri^ry and Information Science, 
Drexel University, Philadelphia, Pa., Septerr.ber, 1979. 

2. Individualized Ins truction for Data Access , Quarterly Report No, 2 
NSF Grant No. DSI 77-26524, School of Library and Information Science, 
Drexel University, Philadelphia, Pa., December, 1979. 

3- A Jto DIALOG - Searchin,^ , Lockheed DIALOG Information Retrieval 

Service, Polo Alto, Ca., 1979. 

Individualized Instruction for Data Access , Quarterl y R eport No, 8 
NSF Grant No. DSI 77-26524, School of Library and Information Science, 
Drexel University, Philadelphia, Pa., Iferch, 1979. 



59 



53 



6, A ppendix 

APPENDIX A: Questionnaires 
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Bibliographic Searching 



Pre -Search Ques tionnaire 



Name: 



Code Word: 



(Note: Your responses to this questionnaire will be confidential 
and will not be made public except in a group or aggregate form 
which will not allow for the identification of any individual's 
responses . ) 



Individualized Instruction for Data Access (IIDA) Project 

Drexel University 
Philadelphia, Pa. 
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We would like to have you give us some information about yourself so that 
we have a better idea of the needs and interests of our users. On the 
following pages you will find several questions. Some call for a written 
answer, others, for a check mark indicating your choice among several 
alternatives. Even if you find some of the questions strange or inappropriate 
it is important that you complete them all. 

1. What is (was) your undergraduate major in college? (Be as specific as 
possible) 



2. Year undergraduate degree received or expected. 

3. Do you have any graduate degrees? Yes: No: 

If yes, what degree(s) in what area(s)? (e.g., M.S. in Psychology) 



4. Please give a brief description, with field and title, of the full-time 
jobs you have held during the last five years. 



5. Please give a brief description, with field and title, of the full-time 
job you would most like td be working at five years from now. 



6. Do you know any computer languages? Yes: No: 



If yes, which one(s)? 



62 



56 



?• How would you describe your computer progranniing skills? 

:Non-exis tant 

:Poor 

:Good enough to get by 

: Quite good 

rExcellent 



8. Have you ever had anyone do on-line computerized bibliographic searching 
for you? 

.Yes: No: 



If yes, roughly how many tines or how often? 



9. Have you, yourself, ever done any on-line searching? 

None: Some: A lot: 



When you have identified a work-related problem, how do you identify the 
information you need to solve the problem? That is, to what extent do you 
rely on the following individuals or resources? 

Asking co-workers: 



not at all very little sometimes a moderate amount considerably 
Asking professional colleagues: 



not at all very little sometimes a moderate amount considerably 
Asking your supervisor: 



not at all very little sometimes a moderate amount considerably 
Going to the library: 



not at all very little sometimes a moderate amount considerably 
Asking the information, retrieval specialist: 



not dt all very little sometimes a moderate amount considerably 
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Solving a research or design problem often requires several steps, some of 
which may be repeated several times before the problem is solved. 



These typically involve: 



a) recognizing the problem exists 

b) defining the problem 

c) breaking the problem up into 

sub-problems 

d) selecting one of the sub-problems 

for "solution" 

e) generating options 

f) selecting an option 

g) implementing the option 

h) evaluation of the result 



At what point(s) in the problem-solving process is(are) bibliographic 
materials most useful? 



At what point(s) have you typically used bibliographic searching? 



At what point(s) can you imagine yourself making use of theiie resources? 



At what point (s) would you recommend to others that they try bibliographic 
searching? 
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Intuition should play very little role in solving problems: 



Strongly agree Agree Uncertain Disagree Strongly disagree 
I do not like using the computer for work-related activities: 

Z • • • m 

* • • • : : 

Strongly agree Agree Uncertain Disagree Strongly disagree 

I believe that every problem should have a clear solution: 



Strongly agree Agree Uncertain Disagree Strongly disagree 

One of the benefits of an education is that it helps you to 
learn hov to so Ive problems : 

' : : : : 

Strongly 2gree Agree Uncertain Disagree Strongly disagree 

There should be a set or routine procedures to follow in 
solving problems: 

• : : : : : 

Strongly agree Agree Uncertain Disagree Strongly disagree 

I believe that using a computer for work-related projects will be: 

complex: : : : : : : : simple 

very quite slightly neither slightly quite very 

boring: : : : : : : : interesting 

very quite slightly neither slightly quite very 

painful: : : : : : : : pleasurable 

very quite slightly neither slightly quite very 

unsuccessful : : : : : • : : successful 

very quite slightly neither slightly quite very 

worthless: : :_ : : : : : valuable 

very quite slightly neither slightly quite very 

frustrating: : : : : : : :not frustrating 

very quite slightly neither slightly quite very 
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Bibliographic Searching 



Intermediate Questionnaire 



Name: 



Code Word: 



(Note: Your responses to this questionnaire will bo confidential 
and will not be made public except in a group or aggregate form 
which will not allow for the identification of any individual's 
responses .) 



Individualized Instruction for Data Access (IIDA) Project 

Drexel University 
Philadeliiiia, Pa. 
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How do you feel about the amount of effor'. you taade during the search? 



very dissatisfied dissatisfied satisfied very satisfied 
"Acrw do you feel about the anount of tine you spent in the process? 



very dissatisfied dissatisfied satisfied very satisfied 
How do you feel abaat the kind of assistance you received fion the system? 



very dissc^tisf ied dissatisfied satisfied very satisfied 
How do you feel about your tofial search experience? 



very dissatisfied dissatisfied satisfied very satisfied 
How do you feel about the items you retrieved? 



very dissatisfied dissatisfied satisfied very satisfied 
How do you feel about the search as a whole? 



totally useless useless useful extremely useful 
How do you feel about the results of the search itself? 



not at all useful moderately useful very useful — I got what I needed* 

Have you ever done any previous research on the topic of your search? 

Yes: No: 

What percentage of the items retrieved wo jld you say were: 

Very useful: 
Useful: ZZZZZ 
Use less : 



Total: 100% 
Would you recommend this system to your friends: 
Yes: No: 
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:se describe your overall impressions of working with computer assisted 
bib.liographic searching. 

Positive Impressions:- (What did the system-do or say that vou liked jnost?) 



Negative .Impressions: (Wliat: did the system do or 'say that you liked least?) 



Please describe any situations where the system did the, vrong thing. 
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Bibliographic Searching 



Post ^Search Ques t ionnaire 



Name: 



Code Word; 



(Note: Your responses to this questionnaire will be confidential 
and will not be made public except in a group or aggregate form 
which will not allow for the identification of any individual's 
responses . ) 



Individualized Instruction for Data Access (IIDA) Project 

Drexel University 
Philadelphia, Pa- 
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How do you fc^el about the amount of effort you made during the s'=:arch? 



ver/ dissatisfied dissatisfied satisfied very satisfied 
How do you feel about the amount of time you spent in the process? 



very dissatisfied dissatisfied satisfi ed very satisfied 
How do you feel about the kind of assistance you received from the system? 



very dissatisfied dissatisfied satisfied very satisfied 
How do you feel about your total search experience? 



very dissatisfied dissatisfied satisfied very satisfied 
How do you feel about the items you retrieved? 



very dissatisfied dissatisfied satisfied very satisfied 
How do you feel about the search as a whole? 



totally useless useless useful extremely useful 
How do you feel about the results of the search itself? 



not at all useful moderately useful very useful — I got wHat I needed 

Have you ever done any previous research on the topic of your search? 

Yes: No: 

What percentage of the items retrieved would you say were: 

Very useful: " 

Useful: 

Useless: 



Total: lOOZ 
Would you recommend this system to your friends: 
Yes: No: 
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Please describe your overall impressions of working with computer assisted 
bib liograf)h ic searching . 

Positive Inpressions: (What did the system do or say that you liked most?) 



Negative Impressions: (\^^lat did the system do or say that you liked least?) 



Please describe any situations where the system did the wrong thing. 



65 

The process of searching was similar to what I had expected: 



strongly agree sgren uncertain disagree strongly disagree 

The search system was characterized by instructions that were easy to 
follow: 



strongly sgree agree uncertain disagree strongly disagree 
The search system vas frustrating to use: 



strongly agree agree uncertain disagree strongly disagree 
The search system was stimulating to use: 



strongly agree agree uncertain disagree strongly disagree 
The use of the search system was easier than expected: 



strongly agree agree uncertain disagree strongly disagree 



How much understanding of the information retrieval system do you feel 
you gained through your experience? 



none very little moderate considerable complete 

Do you feel you have learned enough about searching that you could search on 
your own? 

Yes: No: 



If yes, when did you first decide that you knew enough about the 
process to do it on your own? 



Please explain your reasons for feeling that you could or could not 
search on your own without assistance. 
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